Merging annotations of paginated digital content

ABSTRACT

Features and techniques are disclosed for interacting with paginated digital content, including a multi-purpose tool and an annotation mode. The multi-purpose tool, which may be represented by a graphic (e.g., a movable interactive graphic), can provide access to multiple modes (e.g., copy, define, note, and/or highlight modes) that a user can invoke. The mode invoked determines the functions performed by the tool when interacting with the paginated digital content. The annotation mode, which may be invoked using the multi-purpose tool or independently thereof, can allow a user to create and edit annotations, such as highlights and notes (e.g., sticky notes, margin notes, and/or highlight notes), for paginated digital content. Editing the annotations may include selecting a desired color for the annotation, for example. The annotation mode may also allow a user to intuitively merge and delete annotations previously added to paginated digital content.

RELATED APPLICATIONS

This application is related to: U.S. application Ser. No. ______ filedDec. 31, 2013 (Attorney Docket #BN01.878US) and titled Multi-PurposeTool for Interacting with Paginated Digital Content; U.S. applicationSer. No. ______ filed Dec. 31, 2013 (Attorney Docket #BN01.879US) andtitled Annotation Mode Including Multiple Note Types for PaginatedDigital Content; and U.S. application Ser. No. ______ filed Dec. 31,2013 (Attorney Docket #BN01.834US) and titled Deleting Annotations ofPaginated Digital Content. Each of these applications is hereinincorporated by reference in its entirety.

FIELD OF THE DISCLOSURE

This disclosure relates to electronic computing devices, and moreparticularly, to user interface (UI) techniques for interacting withpaginated digital content on such devices.

BACKGROUND

Electronic computing devices such as tablets, eReaders, mobile phones,smart phones, personal digital assistants (PDAs), and other such devicesare commonly used for displaying digital content. The content may be,for example, an eBook, a web page, an online article or blog, images, amovie or video, a map, just to name a few types. Such devices may alsobe useful for displaying a user interface that allows a user to interactwith one or more applications running on the device. The applicationsmay allow a user to read or browse through paginated digital content,such as electronic books (eBooks), magazines, catalogs, or comics, forexample. The computing devices may also include a touch-sensitivesurface/interface for receiving user input such as a touch screen or atrack pad (e.g., in combination with a non-touch-sensitive display). Theuser may interact with such touch-sensitive electronic computing devicesusing fingers and/or a stylus, for example.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a illustrates an example electronic computing device, configuredin accordance with one or more embodiments of the present invention.

FIGS. 1 b-f illustrate example configuration screen shots of the userinterface of the computing device shown in FIG. 1 a, in accordance withone or more embodiments of the present invention.

FIG. 2 a illustrates a block diagram of an electronic computing device,configured in accordance with an embodiment of the present invention.

FIG. 2 b illustrates a block diagram of a communication system includingthe electronic computing device of FIG. 2 a, configured in accordancewith an embodiment of the present invention.

FIGS. 3 a-m collectively illustrate an example multi-purpose tool for anelectronic computing device, configured in accordance with one or moreembodiments of the present invention.

FIGS. 4 a-d illustrate an example copy mode of an electronic computingdevice, configured in accordance with an embodiment of the presentinvention

FIGS. 5 a-d illustrate an example define mode of an electronic computingdevice, configured in accordance with an embodiment of the presentinvention.

FIGS. 6 a-d illustrate an example highlight mode of an electroniccomputing device, configured in accordance with an embodiment of thepresent invention.

FIGS. 7 a-d illustrate creating and editing a sticky note using anexample note mode of an electronic computing device, in accordance withan embodiment of the present invention.

FIGS. 8 a-h illustrate creating and editing a margin note using anexample note mode of an electronic computing device, in accordance withan embodiment of the present invention.

FIGS. 9 a-d illustrate creating and editing a highlight note using anexample note mode of an electronic computing device, in accordance withan embodiment of the present invention.

FIGS. 10 a-d illustrate merging sticky notes added to paginated digitalcontent, in accordance with an embodiment of the present invention.

FIGS. 11 a-h illustrate merging highlights including highlight notesadded to paginated digital content, in accordance with an embodiment ofthe present invention.

FIGS. 12 a-d illustrate deleting annotations added to paginated digitalcontent, in accordance with one or more embodiments of the presentinvention.

FIGS. 12 e-h illustrate undoing the annotation deletions performed inFIGS. 12 a-d, in accordance with one or more embodiments of the presentinvention.

FIGS. 13 a-c illustrate example filtering and searching functions for anannotation mode of an electronic computing device, in accordance withone or more embodiments of the present invention.

FIG. 14 illustrates a method for providing a multi-purpose tool in anelectronic computing device, in accordance with one or more embodimentsof the present invention.

FIG. 15 illustrates a method for providing an annotation mode in anelectronic computing device, in accordance with one or more embodimentsof the present invention.

DETAILED DESCRIPTION

Features and techniques are disclosed for interacting with paginateddigital content, including a multi-purpose tool and an annotation mode.The multi-purpose tool, which may be represented by a graphic (e.g., amovable and/or interactive graphic), can provide access to multiplemodes (e.g., copy, define, note, and/or highlight modes) that a user caninvoke. The mode invoked determines the functions performed by the toolwhen interacting with the paginated digital content. The annotationmode, which may be invoked using the multi-purpose tool or independentlythereof, can allow a user to create and edit annotations, such ashighlights and notes (e.g., sticky notes, margin notes, and/or highlightnotes), for paginated digital content. For example, a user may create asticky note by tapping on the paginated content when the annotation modeis invoked, and the sticky note may be converted to a margin note (e.g.,by toggling between the note types) or a highlight note (e.g., bydragging and dropping the sticky note onto a previously createdhighlight). Editing the annotations may include selecting the desiredcolor for the annotation, for example. The annotation mode may alsoallow a user to intuitively merge and delete annotations previouslyadded to paginated digital content. Numerous other configurations andvariations will be apparent in light of this disclosure.

General Overview

As previously explained, electronic computing devices such as tablets,eReaders, and smart phones are commonly used for displaying userinterfaces and digital content. The user of such devices can typicallyconsume the displayed digital content with relative ease. In someinstances, the content being consumed, such as electronic books(eBooks), magazines, catalogs, comics, or other digital documents, maybe divided into individual pages. Some applications or user interfacesof the devices provide the ability to interact with such paginatedcontent, including copying text from the content, defining words withinthe content, or annotating the content. However, such conventionaltechniques require the user to first select the text of interest andthen select the function of interest. In addition, such conventionaltechniques typically allow only limited functionality with respect toannotating paginated digital content, which may lead to a diminisheduser experience.

Thus, and in accordance with one or more embodiments of the presentinvention, features and techniques are disclosed for interacting withpaginated digital content presented/displayed by an electronic computingdevice. The features and techniques may include, in some embodiments, amulti-purpose tool that can provide access to multiple modes (e.g., copymode, define mode, highlight mode, note mode, etc.) that a user caninvoke. The mode invoked determines the functions performed by the toolwhen interacting with the paginated digital content. The multi-purposetool may be represented, for instance, by a graphic (e.g., a movableand/or interactive graphic) that provides access to the modes usingvarious suitable input to invoke a desired mode (e.g., tap/select on thetool graphic to expand then tap/select desired mode, tap/select on thetool graphic to expand then swipe to desired mode, swipe in direction ofdesired mode from the tool graphic, etc.). For example, themulti-purpose tool may be used in a highlight mode that, when invoked,allows a user to intuitively swipe over text of the paginated digitalcontent to add highlighting to the text. In such an example, the colorof the highlighting may be simultaneously selected when invoking thehighlight mode, as will be apparent in light of this disclosure. Inanother example, the multi-purpose tool may be used in a define modethat, when invoked, allows a user to intuitively tap on or select thedesire word to be defined, causing the definition to be presented. Otherexample modes accessible from the multi-purpose tool will be apparent inlight of this disclosure.

The features and techniques for interacting with paginated digitalcontent may also include, in some embodiments, an annotation mode thatallows a user to create and edit annotations for the paginated digitalcontent. Annotations may include, for example, highlighting or notes.Thus, in some embodiments, all or portions of the highlight and notemodes (as variously described herein) may be integrated into a singleannotation mode, as will be apparent in light of this disclosure.Highlighting generally refers to a selection of content (e.g., textualcontent) affected with a distinct coloration of that content. A notegenerally refers to user generated content that can be typed orhand-written or otherwise provided over or next to existing content. Aspreviously described, a user may be able to create highlighting bydragging over text of paginated digital content when a highlight mode isinvoked (or more generally, in some embodiments, when an annotation modeis invoked). After the highlighting has been added to the paginatedcontent, the user may be able to edit the highlighting, in someembodiments, by changing the color of the highlighting and/or byexpanding/contracting the textual coverage of the highlighting, forexample.

In some embodiments, a user may be able to create a sticky note bytapping on or selecting paginated digital content when a note mode isinvoked (or more generally, in some embodiments, when an annotation modeis invoked). In some such embodiments, a sticky note may be created onthe paginated digital content at the tap or selection location. Stickynotes, in some embodiments, may be represented by a graphic (e.g., amovable and/or interactive graphic) that provides access to the contentsof the sticky note. In some such embodiments, after a sticky note hasbeen created, the user may be able to edit the contents of the stickynote using, for example, an edit box that is presented when the stickynote icon is tapped on or selected. The edit box may also allow the userto change the color of the sticky note or otherwise edit the stickynote, as will be apparent in light of this disclosure.

In some embodiments, a user may be able to create a margin note from apreviously created sticky note by converting the sticky note to a marginnote. In some such embodiments, the sticky note edit box may include atoggle tool to allow the user to switch between the note types. Marginnotes may differ from sticky notes, in some instances, in that marginnotes always present/show the contents of the note whereas sticky notesmay be represented by a graphic/icon that can be tapped on or selected(or otherwise interacted with) to present/show its contents. Stickynotes and margin notes, in some embodiments, may be capable of beingmoved anywhere on the paginated content. In some embodiments, a user maybe able to create a highlight note by dragging and dropping a previouslycreated sticky note onto previously created highlighting. In some suchembodiments, the highlight note retains the color of the sticky note anddoes not inherit the color of its associated highlight. In someembodiments, a highlight note may be detached from its associatedhighlight to change it back to a sticky note by, for example, draggingand dropping the highlight note away from the highlighting. Note that insome embodiments, a margin note can be positioned over highlighting(compared to sticky notes, in some embodiments, where attempting toposition a sticky note over highlighting may instead convert the stickynote to a highlight note as previously described).

In some embodiments, the annotation mode may also allow a user tofurther interact with annotations (e.g., highlights and notes) added topaginated digital content. Such interactions may include intuitivelymerging and deleting annotations as well as being able to view afiltered list of and search the annotations. In some embodiments,merging annotations may include dragging and dropping one annotationonto (or over) another to merge the two annotations together. Forexample, in some embodiments, sticky notes can be merged by performing apress-and-hold on a sticky note to grab it, dragging the grabbed stickynote to another sticky note, and then dropping the grabbed sticky noteonto the other sticky note to merge the two notes together. In anotherexample, in some embodiments, highlights can be merged together bydragging/expanding and dropping one highlight over another to merge thetwo highlights together. In some embodiments, deleting annotations mayinclude tapping or selecting a deletion graphic associated with theannotation or performing a press-and-hold then flick input on theannotation (e.g., to throw the annotation off of the screen and deleteit). Other suitable annotation merge and delete features and techniqueswill be apparent in light of this disclosure. In some embodiments,annotation merges and/or deletions may be able to be undone using anundo input (e.g., shake input or a Z-shaped gesture input), as will alsobe apparent in light of this disclosure.

In some embodiments, the features for interacting with paginated digitalcontent (e.g., a multi-purpose tool, an annotation mode,merging/deleting/filtering/searching annotations functionality, etc.)may be configured at a global level (e.g., based on the UI settings ofthe device) and/or at an application level (e.g., based on the specificapplication being used to interact with the paginated digital content).For example, one or more of the features may be configured to be enabledfor some applications and disabled for others. To this end, one or moreof the features may be user-configurable, hard-coded, or somecombination thereof (e.g., where some aspects are user-configurable andothers are hard-coded), as will be apparent in light of this disclosure.Further, one or more of the features for interacting with paginateddigital content may be included initially with the UI (or operatingsystem) of an electronic computing device or be included in a separateprogram/service/application configured to interface with the UI of sucha device to incorporate the functionality of such features as variouslydescribed herein. In some embodiments, one or more of the features forinteracting with paginated digital content may come in the form of acomputer program product (one or more non-transient computer readablemediums) that includes a plurality of instructions non-transientlyencoded thereon, that when executed by one or more processors, cause aprocess to be carried out that includes the functionality of one or moreof the features as variously described herein.

As will be apparent in light of this disclosure, the features andtechniques described herein (e.g., a multi-purpose tool, an annotationmode, merging/deleting/filtering/searching annotations functionality,etc.) may be used with any paginated or similarly divided digitalcontent such as, for example, eBooks, magazines, catalogs, comics,documents, notes, presentations, lecture materials, photo albums, and/orany other suitable paginated or similarly divided digital content. Aswill also be apparent, the features for interacting with paginateddigital content may be used on any suitable electronic computing device,such as various smartphones, tablets, and eReaders. In such devicesincluding a touch-sensitive UI, user input may be referred to as contactor user contact for ease of reference. However, direct and/or proximatecontact (e.g., hovering within a few centimeters of the touch-sensitivesurface) may be used to provide user input to the device, depending onthe specific touch-sensitive surface/interface being used. In otherwords, in some embodiments, a user need not even physically touch thedevice or its touch-sensitive surface/interface to provide input. Alsonote that the user contact (whether direct or proximate) may be providedby the user's hand (or another suitable body part), or a stylus (or someother suitable implement), for example. In some embodiments, user inputmay be provided in various other ways, such as using mouse gestures fordevices that accept mouse input (e.g., desktop, laptop, tabletcomputers) or through voice commands or sounds, or through any othersuitable method. Numerous variations and configurations will be apparentin light of this disclosure.

Computing Device and Configuration Examples

FIG. 1 a illustrates an example electronic computing device configuredin accordance with one or more embodiments of the present invention. Thedevice (or one or more of its components) may include numerous featuresfor interacting with paginated digital content, such as a multi-purposetool, an annotation mode, a merge annotations function, and/or a deleteannotations function as variously described herein. The device could be,for example, a tablet such as the NOOK® tablet by Barnes & Noble®. In amore general sense, the device may be any computing device capable ofdisplaying digital content, such as a smart phone, eReader, tabletcomputer, laptop, or desktop computer, for example. In some instances,the computing device may be touch-sensitive and include a touch screendisplay or a non-touch-sensitive display screen that can be used inconjunction with a touch-sensitive surface/interface, such as a trackpad. As will be appreciated in light of this disclosure, the presentdisclosure is not intended to be limited to any particular kind or typeof computing device. For ease of description, an example computingdevice is provided herein with touch screen technology.

As can be seen with the example embodiment shown in FIG. 1 a, the devicecomprises a housing/frame that includes a number of hardware featuressuch as a power button and a press-button (sometimes called a homebutton herein). A touch screen based user interface (UI) is alsoprovided (although a computing device running one or more of thefeatures for interacting with paginated digital content as describedherein need not be touch-sensitive), which in this example embodimentincludes a quick navigation menu having six main categories to choosefrom (Home, Library, Shop, Search, Light, and Settings) and a status barthat includes a number of icons (a night-light icon, a wireless networkicon, and a book icon), a battery indicator, and a clock. Otherembodiments may have fewer or additional such UI touch screen controlsand features, or different UI touch screen controls and featuresaltogether, depending on the target application of the device. Any suchgeneral UI touch or non-touch controls and features can be implementedusing any suitable conventional or custom technology, as will beappreciated.

Although the computing device shown in FIG. 1 a uses a touch screendisplay, other touch-sensitive embodiments may include a non-touchscreen and a touch-sensitive surface such as a track pad, or atouch-sensitive housing configured with one or more acoustic sensors,etc. As previously described, the computing device need not betouch-sensitive and may receive input from physical buttons, directionalpads, joysticks, mouse pointing devices, and physical keyboards, just toname a few other input device examples. Continuing with the examplecomputing device shown in FIG. 1 a, the power button can be used to turnthe device on and off, and may be used in conjunction with a touch-basedUI control feature that allows the user to confirm a given powertransition action request (e.g., such as a slide bar or tap pointgraphic to turn power off). In this example device, the home button is aphysical press-button that can be used to display the device's homescreen when the device is awake and in use. Note that the buttons asvariously described herein may be physical, virtual, or some combinationthereof, depending upon the embodiment. The home button (or otherbuttons) may also be used in conjunction with the features as disclosedherein to, for example, exit the annotation mode after it has beeninvoked. Numerous other configurations and variations will be apparentin light of this disclosure, and the present disclosure is not intendedto be limited to any particular set of control features or device formfactor.

Continuing from FIG. 1 a, the user can access a configuration sub-menu,such as the Multi-Purpose Tool configuration sub-menu shown in FIG. 1 cor the Annotation Mode configuration sub-menu shown in FIG. 1 d, bytapping or otherwise selecting the Settings option in the quicknavigation menu, which causes the device in this embodiment to displaythe general sub-menu shown in FIG. 1 b. From this general sub-menu theuser can select any one of a number of options, including one designatedUser Interface (UI) in this specific example case. Selecting thissub-menu item (with, for example, an appropriately placed screen tap)may cause the configuration sub-menu of FIG. 1 c or 1 d to be displayed,in accordance with an embodiment. In other example embodiments,selecting the User Interface (UI) option may present the user with anumber of additional sub-options, one of which may include an eReaderoption, which may then be selected by the user so as to cause theconfiguration sub-menu of FIG. 1 c or 1 d to be displayed. Any number ofsuch menu schemes and nested hierarchies can be used, as will beappreciated in light of this disclosure. In other embodiments, thefeatures for interacting with paginated digital content as variouslydescribed herein (e.g., a multi-purpose tool, an annotation mode,merging/deleting/filtering/searching annotations functionality, etc.)may be hard-coded such that no configuration is needed or otherwisepermitted. The degree of hard-coding versus user-configurability canvary from one embodiment to the next, and the present disclosure is notintended to be limited to any particular configuration scheme of anykind, as will be apparent in light of this disclosure.

As will be appreciated, the various UI control features and sub-menusdisplayed to the user are implemented as UI touch screen controls inthis example embodiment. Such UI touch screen controls can be programmedor otherwise configured using any number of conventional or customtechnologies. In general, the touch screen translates one or moretouches (whether direct or proximate and whether made by a user's hand,a stylus, or some other suitable implement) in a particular location(s)into an electrical signal, which is then received and processed by theunderlying operating system (OS), system software, and circuitry(processor, etc.) of the computing device. In some instances, note thatthe user need not actually physically touch the touch-sensitivesurface/interface to provide user input (e.g., when the touch-sensitivesurface/interface recognizes hovering input). In embodiments where thecomputing device is not-touch-sensitive, input may be provided using amouse, joystick, or directional pad, and one or more buttons, forexample, to provide input similar to touching a touch screen. In someembodiments, the UI may allow other user interactions, such asvoice-commands. Additional example details of the underlying OS andcircuitry in accordance with some embodiments will be discussed in turnwith reference to FIG. 2 a. As previously explained, in some cases, thefeatures for interacting with paginated digital content as variouslydescribed herein (e.g., a multi-purpose tool, an annotation mode,merging/deleting/filtering/searching annotations functionality, etc.)may be automatically configured by the specific UI or application beingused. In these instances, the features need not be user-configurable(e.g., if the features are hard-coded or are otherwise automaticallyconfigured).

Continuing with FIG. 1 b, once the Settings sub-menu is displayed, theuser can then select the User Interface (UI) option. In response to sucha selection, the Multi-Purpose Tool configuration sub-menu shown in FIG.1 c or the Annotation Mode configuration sub-menu shown in FIG. 1 d canbe provided to the user. Alternatively, the Annotation Modeconfiguration sub-menu shown in FIG. 1 d may be accessed through theMulti-Purpose tool sub-menu using, for example, a Configure buttoncorresponding to an Annotation Mode, Highlight Mode, or Note Mode (e.g.,as shown in FIG. 1 c). In the example case shown in FIG. 1 c, theMulti-Purpose Tool configuration sub-menu includes a UI check box thatwhen checked or otherwise selected by the user, effectively enables thetool (shown in the Enabled state); unchecking the box disables the tool.Other embodiments may have the multi-purpose tool always enabled, orenabled by a switch or button, for example. In some instances, themulti-purpose tool may be automatically enabled in response to anaction, such as when paginated digital content is displayed (orotherwise presented to the user) or a specific application is launched,for example. As previously described, the user may be able to configuresome or all of the features with respect to the multi-purpose tool, soas to effectively give the user a say in, for example, which modes themulti-purpose tool provides access to, configuration of the accessiblemodes, how to invoke the accessible modes from the multi-purpose tool,and/or other suitable configuration options as will be apparent in lightof this disclosure, if so desired.

In the example case shown in FIG. 1 c, once the multi-purpose tool isenabled, the user can configure various options related to this exampletool. As shown, the left side of this example settings screen shotincludes multiple configurable options. Such options include the abilityto select how many modes the tool can provide access to (e.g., byselecting the check boxes under each mode) and what specific modes thoseare. In this example configuration, the tool includes four potentialmodes (Modes 1-4) and all four are shown in an enabled state (each modecheck box is selected). In other embodiments, the multi-purpose tool mayprovide access to more modes, such as eight potential modes, or lessmodes, such as two potential modes. As can be seen in this exampleconfiguration, Mode 1 is set as a Highlight Mode, Mode 2 is set as aNote Mode, Mode 3 is set as a Copy Mode, and Mode 4 is set as a DefineMode, each of which are discussed in more detail herein. The modes(Modes 1-4) have been set using the respective drop-down menus and eachmode includes a respective selectable Configure button that allows auser to further configure the set modes. Other modes may include anannotation mode, a lookup mode, a cut mode, and a delete mode, as willbe apparent in light of this disclosure. An example of a settingssub-menu showing additional configuration options for an annotation modeis shown in FIG. 1 d and will be discussed in more detail below.

The left side of the settings screen shot also includes a Configure ModeSelection Input button, that may allow a user to configure how the modes(e.g., Modes 1-4 in this example) are selected using the multi-purposetool. Such selection input options may include being able to swipe inthe direction of the desired mode, selecting (or tapping on) the tool toshow the modes available and then selecting (or tapping on) the desiredmode, or any other suitable input as will be apparent in light of thisdisclosure. For example, the selection input may be set such that Mode 1can be invoked by swiping rightward from the multi-purpose tool, Mode 2can be invoked by swiping leftward from the tool, Mode 3 can be invokedby swiping upward from the tool, and Mode 4 can be invoked by swipingdownward from the tool. The right side of the example settings screenshot including the Applications section will be discussed in more detailbelow.

In the example case shown in FIG. 1 d, once the annotation mode isenabled, the user can configure various options related to this examplemode. In some embodiments, the mode may be set as one of the modesaccessible from the multi-purpose tool (e.g., as shown in FIG. 1 c);however, in other embodiments, the annotation mode may also be its ownseparate feature external to the multi-purpose tool. As will be apparentin light of this disclosure, the annotation mode may also include thefunctionality of the highlight and note modes as variously describedherein. Therefore, the annotation mode may allow a user to highlightand/or add notes to paginated digital content. The annotation mode inthis example case includes three different note types (Sticky Notes,Margin Notes, and Highlight Notes), which can be enabled/disabled usingthe respective boxes under the Note Types section of the sub-menu inFIG. 1 d. This example settings sub-menu also allows a user to set theInitial Annotation Color (e.g., the initial color for highlightingand/or for created notes) using the corresponding drop-down menu (set asYellow in this example embodiment). The user may also select theConfigure Colors button to further configure color options, such aswhich colors are available to be selected when highlighting and/orcreating notes on paginated digital content. The three different notetypes (sticky, margin, and highlight notes) are discussed in more detailherein.

The example Annotation Mode setting screen shot shown in FIG. 1 d alsoincludes two selectable buttons to Configure Merging and ConfigureDeleting for the annotation mode. In this example case, selection of theConfigure Merging button may cause the example Annotation Mode MergingAnnotations settings sub-menu to be presented as shown in FIG. 1 e, andselection of the Configure Deleting button may cause the exampleAnnotation Mode: Deleting Annotations settings sub-menu to be presentedas shown in FIG. 1 f. However, such sub-menus may be accessed in anothersuitable manner, and in some embodiments, the merging annotationsfunctionality and deleting annotations functionality (as will bediscussed in more detail herein) may be a feature of or separate from anannotation mode. As will be apparent in light of this disclosure, themerging annotations and deleting annotations functionality may beapplicable to highlighting and/or adding notes to paginated digitalcontent.

FIG. 1 e shows the Merge Annotations functionality as enabled in thisexample sub-menu screen shot. As can also be seen, the user can AllowMerging For Sticky Notes, Highlights, and/or Highlight Notes byselecting the note type's corresponding check box (shown with mergingallowed/enabled for all three). Note that margin notes cannot be merged,as will be apparent in light of this disclosure. In this examplesettings sub-menu, the user can also select the Primary Annotation whenmerging annotations. This configuration option allows a user toconfigure whether the annotation being dragged (e.g., the highlight ornote being dragged) or the annotation dragged to (e.g., the highlightbeing dragged to or the note being dropped on) dictates the color of theannotation, the location of the annotation, and/or the initial contentof a resulting note associated with the merged annotations. Such afeature of the merging annotations functionality will be discussed inmore detail herein, and the example screen shot shown in FIG. 1 e isprovided to illustrate that such a feature may be user-configurable(however, such a feature may also be hard-coded).

FIG. 1 f shows the Delete Annotations functionality as enabled in thisexample sub-menu screen shot. As can also be seen, the user can select aDelete Input using the corresponding drop-down menu. The selected deleteinput shown is a Press-and-Hold then Flick Gesture. Other delete inputmay include selecting an associated deletion graphic related to anannotation (e.g., an “X” or garbage can icon), or other suitable inputas will be apparent in light of this disclosure. The delete annotationsfunctionality may also allow for more than one delete input, such asallowing both the press-and-hold then flick gesture input and selectionof an associated deletion graphic related to an annotation to providedelete input. The Delete Annotations sub-menu shown in this example casealso allows a user to select if Undo Delete is Allowed (shown enabled).The undo delete feature allows a user to undo the deletion of anannotation using, for example, the input selected in the correspondingdrop-down menu. As can be seen, the undo input is set as a Shake Input(and can be further configured using the Configure Input button), whichmay be invoked by shaking the computing device (as will be discussed inmore detail herein). Other undo delete input may include a Ctrl+Zcommand, inputting a Z-shaped gesture, selecting an undo button, orother suitable input as will be apparent in light of this disclosure.The delete annotations functionality may also allow for more than oneundo delete input, such as a shake input and inputting a Z-shapedgesture, for example. The delete annotations feature also includes aPresent Undo Dialog configuration option (shown enabled), which presentsa selectable option to undo the deletion of an annotation. Such a dialogmay be presented only after an annotation has been deleted or only whenone or more annotation deletions can be undone (as will be discussed inmore detail herein), and the feature may be further configured using theConfigure Dialog button shown.

The right side of the example settings sub-menu screen shots shown inFIGS. 1 c-f include Applications sections that allow a user toindividually enable the features for interacting with paginated digitalcontent (e.g., a multi-purpose tool, an annotation mode,merging/deleting/filtering/searching annotations functionality, etc.)for specific applications, in one or more embodiments. Such aconfiguration feature may be helpful, for instance, in a tablet orlaptop or other multifunction computing device that can executedifferent applications (as opposed to a computing device that is more orless dedicated to a particular application). As previously explained, insome embodiments, the features for interacting with paginated digitalcontent may be application specific or configured at an applicationlevel. As can be seen in the example screen shots shown in FIGS. 1 c-f,the features are enabled for the eBook Reader, Office Suite,Magazine/Catalog Reader, and Notebook applications. For completeness ofdescription, the features are disabled for the Comic Reader and PictureGallery, in this example case, such that the features may not even bepresented or available in the applications. In some embodiments, one ormore of the features may be later enabled from this settings screen orenabled from the specific application itself.

In some embodiments, one or more of the features for interacting withpaginated digital content (e.g., a multi-purpose tool, an annotationmode, merging/deleting/filtering/searching annotations functionality,etc.) may be related or tied to one or more specific applications of thedevice's UI (or operating system), such that the feature(s) is onlyavailable, active, or running when such an application is available,active, or running. For example, the multi-purpose tool may only beavailable, active, or running when an eReader application is activeand/or displayed, which may save a computing device's memory and/orpower consumption. Any number of applications or device functions maybenefit from the features for interacting with paginated digital contentas provided herein, whether user-configurable or not, and the presentdisclosure is not intended to be limited to any particular applicationor set of applications.

As can be further seen in FIGS. 1 c-f, a back button arrow UI controlfeature may be provisioned on the touch screen for any of the menusprovided, so that the user can go back to the previous menu, if sodesired. Note that configuration settings provided by the user can besaved automatically (e.g., user input is saved as selections are made orotherwise provided). Alternatively, a save button or other such UIfeature can be provisioned, which the user can engage as desired. Again,while FIGS. 1 c-f show user-configurability, other embodiments may allowfor a different configuration scheme or they may not allow for any suchconfiguration, wherein the various features provided are hard-coded orotherwise provisioned by default. The degree of hard-coding versususer-configurability can vary from one embodiment to the next, and thepresent disclosure is not intended to be limited to any particularconfiguration scheme of any kind.

Architecture

FIG. 2 a illustrates a block diagram of a computing device configured inaccordance with an embodiment of the present invention. As can be seen,this example device includes a processor, memory (e.g., RAM and/or ROMfor processor workspace and storage), additional storage/memory (e.g.,for content), a communications module, a touch screen, and an audiomodule. A communications bus and interconnect is also provided to allowinter-device communication. Other typical componentry and functionalitynot reflected in the block diagram will be apparent (e.g., battery,co-processor, etc.). Further note that although a touch screen displayis provided, other touch-sensitive embodiments may include a non-touchscreen and a touch-sensitive surface such as a track pad, or atouch-sensitive housing configured with one or more acoustic sensors,etc. In this manner, a non-touch-sensitive computing device can become atouch-sensitive computing device by adding an interfacingtouch-sensitive component. However, as previously explained, someembodiments may be non-touch-sensitive. The principles provided hereinequally apply to any such computing device. For ease of description,examples are provided with touch screen technology.

The touch-sensitive surface (touch-sensitive display or touch screen, inthis example) can be any device that is configured with user inputdetecting technologies, whether capacitive, resistive, acoustic, activeor passive stylus, and/or other input detecting technology. The screendisplay can be layered above input sensors, such as a capacitive sensorgrid for passive touch-based input (e.g., with a finger or passivestylus in the case of a so-called in-plane switching (IPS) panel), or anelectro-magnetic resonance (EMR) sensor grid (e.g., for sensing aresonant circuit of the stylus). In some embodiments, the touch screendisplay can be configured with a purely capacitive sensor, while inother embodiments the touch screen display may be configured to providea hybrid mode that allows for both capacitive input and active stylusinput. In any such embodiments, a touch screen controller may beconfigured to selectively scan the touch screen display and/orselectively report contacts detected directly on or otherwisesufficiently proximate to (e.g., within a few centimeters) the touchscreen display. The proximate contact may include, for example, hoveringinput used to cause location specific input as though direct contactwere being provided on a touch-sensitive surface (such as a touchscreen). Numerous touch screen display configurations can be implementedusing any number of known or proprietary screen based input detectingtechnology.

Continuing with the example embodiment shown in FIG. 2 a, the memoryincludes a number of modules stored therein that can be accessed andexecuted by the processor (and/or a co-processor). The modules includean operating system (OS), a user interface (UI), and a powerconservation routine (Power). The modules can be implemented, forexample, in any suitable programming language (e.g., C, C++, objectiveC, JavaScript, custom or proprietary instruction sets, etc.), andencoded on a machine readable medium, that when executed by theprocessor (and/or co-processors), carries out the functionality of thedevice, including one or more of the features for interacting withpaginated digital content (e.g., a multi-purpose tool, an annotationmode, merging/deleting/filtering/searching annotations functionality,etc.) as variously described herein. The computer readable medium maybe, for example, a hard drive, compact disk, memory stick, server, orany suitable non-transitory computer/computing device memory thatincludes executable instructions, or a plurality or combination of suchmemories. Other embodiments can be implemented, for instance, withgate-level logic or an application-specific integrated circuit (ASIC) orchip set or other such purpose built logic, or a microcontroller havinginput/output capability (e.g., inputs for receiving user inputs andoutputs for directing other components) and a number of embeddedroutines for carrying out the device functionality. In short, thefunctional modules can be implemented in hardware, software, firmware,or a combination thereof.

The processor can be any suitable processor (e.g., Texas InstrumentsOMAP4, dual-core ARM Cortex-A9, 1.5 GHz), and may include one or moreco-processors or controllers to assist in device control. In thisexample case, the processor receives input from the user, includinginput from or otherwise derived from the power button, home button, andtouch-sensitive surface. In other embodiments, the processor may beconfigured to receive input from other input devices, such as from amouse or keyboard, for example, to determine if such devices are beingused to provide input. The processor can also have a direct connectionto a battery so that it can perform base level tasks even during sleepor low power modes. The memory (e.g., for processor workspace andexecutable file storage) can be any suitable type of memory and size(e.g., 256 or 512 Mbytes SDRAM), and in other embodiments may beimplemented with non-volatile memory or a combination of non-volatileand volatile memory technologies. The storage (e.g., for storingconsumable content and user files) can also be implemented with anysuitable memory and size (e.g., 2 GBytes of flash memory).

The display can be implemented, for example, with a 7 to 9 inch1920×1280 IPS LCD touchscreen touch screen, or any other suitabledisplay and touch-sensitive interface technology. The communicationsmodule can be, for instance, any suitable 802.11b/g/n WLAN chip or chipset, which allows for connection to a local network so that content canbe downloaded to the device from a remote location (e.g., contentprovider, etc., depending on the application of the display device). Insome specific example embodiments, the device housing or frame thatcontains all the various componentry measures about 7″ to 9″ high byabout 5″ to 6″ wide by about 0.5″ thick, and weighs about 7 to 8 ounces.Any number of suitable form factors can be used, depending on the targetapplication (e.g., laptop, smart phone, etc.). The device may, forexample, smaller for smart phone and eReader applications and larger fortablet computer applications.

The operating system (OS) module can be implemented with any suitableOS, but in some example embodiments is implemented with Google AndroidOS or Linux OS or Microsoft OS or Apple OS. The power management (Power)module can be configured as typically done, such as to automaticallytransition the device to a low power consumption or sleep mode after aperiod of non-use. A wake-up from that sleep mode can be achieved, forexample, by a physical button press and/or a touch screen swipe or otheraction. The audio module can be configured, for example, to speak orotherwise aurally present a selected eBook or other textual content. Insome example cases, if additional space is desired, for example, tostore digital books or other content and media, storage can be expandedvia a microSD card or other suitable memory expansion technology (e.g.,32 GBytes, or higher). The UI module can be, for example, based on touchscreen technology, and the various example screen shots and exampleuse-cases shown in FIGS. 1 a-f and FIGS. 3 a-13 c (including allintervening Figures), in conjunction with the methodologies demonstratedin FIGS. 14 and 15, which will be discussed in turn.

Client-Server System

FIG. 2 b illustrates a block diagram of a communication system includingthe electronic computing device of FIG. 2 a configured in accordancewith an embodiment of the present invention. As can be seen, the systemgenerally includes an electronic computing device that is capable ofcommunicating with a server via a network/cloud. In this exampleembodiment, the computing device may be, for example, an eReader, asmart phone, a laptop, a tablet computer, a desktop computer, or anyother suitable computing device. The network/cloud may be a publicand/or private network, such as a private local area network operativelycoupled to a wide area network such as the Internet. In this exampleembodiment, the server may be programmed or otherwise configured toreceive content requests from a user via the computing device and torespond to those requests by providing the user with requested orotherwise recommended content. In some such embodiments, the server maybe configured to remotely provision one or more of the features asvariously provided herein for interacting with paginated digital content(e.g., a multi-purpose tool, an annotation mode,merging/deleting/filtering/searching annotations functionality, etc.) tothe computing device (e.g., via JavaScript or other browser basedtechnology). In other embodiments, portions of the methodology may beexecuted on the server and other portions of the methodology may beexecuted on the device. Numerous server-side/client-side executionschemes can be implemented to facilitate the features as disclosedherein for interacting with paginated digital content in accordance withone or more embodiments, as will be apparent in light of thisdisclosure.

Multi-Purpose Tool and Example Modes

FIGS. 3 a-m collectively illustrate an example multi-purpose tool for anelectronic computing device, configured in accordance with one or moreembodiments of the present invention. As can be seen in FIG. 3 a, theelectronic computing device is shown having a housing/frame thatincludes a touch screen for displaying content and allowing user input(whether direct or proximate). The touch screen in FIG. 3 a isdisplaying a page of an eBook, and more specifically, page 10 of theeBook “Great Expectations.” The eBook may be displayed or otherwisepresented using an eReader application, or some other suitableapplication or program. Although the multi-purpose tool (and otherfeatures for interacting with paginated digital content) is beingillustrated on a touch-sensitive device having a touch screen, othertouch-sensitive devices may include a non-touch display screen that canbe used in conjunction with a touch-sensitive surface/interface, such asa track pad, as previously described. As was also previously described,the multi-purpose tool (and other features for interacting withpaginated digital content) may be used on a non-touch-sensitivecomputing device; however, a touch-sensitive computing device is usedherein for ease of description.

As will be apparent in light of this disclosure, and in accordance withone or more embodiments, the multi-purpose tool can provide access tomultiple modes to allow a user to invoke one of the modes using thetool. Once a mode has been invoked, a function can be performeddetermined by the invoked mode in response to user input performed onthe paginated digital content. Such techniques are described herein withrespect to a copy mode, define mode, highlight mode, and note mode forillustrative purposes; however, the present disclosure is not intendedto be limited to the specific modes or corresponding functionality asvariously described herein. Note that the highlight mode and note mode,and corresponding functionality of each mode, may be referred tocollectively herein as an annotation mode. Also note that in someembodiments, the modes described herein may be separate from themulti-purpose tool, such that they can be invoked in another manner.Further note that the multi-purpose tool may be configured to lock thepage currently being presented/displayed upon the invocation of one ormore modes to allow for the input to be recognized. For example, if themulti-purpose tool is being used by an eReader application that allows auser to turn pages using swipe gestures, and one of the modes uses aswipe or drag gesture input to perform a function (other than turningthe page), then the multi-purpose tool may be configured to lock thecurrently presented/displayed page, to allow the appropriate function tobe performed on that page. In some embodiments, the multi-purpose toolmay be configured to allow page turning input, even when a mode isinvoked. In some such embodiments, a user may be able to swipe from theright and left edges of the computing device (e.g., a smart phone ortablet) to turn pages of the paginated digital content. In such cases,because the user is swiping over white margin space, and not swipingover text, the multi-purpose tool (and/or an invoked mode) may determinethat a page turn is desired based on the input (e.g., instead ofhighlighting the text when the highlight mode is invoked).

FIG. 3 a shows a multi-purpose tool graphic at the bottom center of thepaginated digital content on the touch screen. As previously described,the multi-purpose tool may be represented by a graphic in someembodiments, and in this example embodiment, is represented with amovable interactive graphic. The tool graphic can be used to invokemodes accessible through the multi-purpose tool. In this exampleembodiment, the tool graphic is configured to be provisioned at thelocation seen in FIG. 3 a by default for such a portrait layout/displayand to be provisioned on the bottom center of a the left most page in amulti-page layout (e.g., a two page magazine-style layout that may beused when the device is in a landscape orientation). Other embodimentsmay provision the tool graphic in different locations by default and insome embodiments, a user may be able to configure the default graphiclocation. In some embodiments, the previous position of the tool graphicmay be remembered, even when opening new paginated digital content. Sucha default tool graphic location may be reset every time that paginateddigital content is opened, for example. Note that the multi-purpose toolgraphic is shown is provided for illustrative purposes only and thepresent disclosure is not intended to be limited to, for example, theshape, size, or colors shown.

FIGS. 3 b-3 d illustrate moving the multi-purpose tool graphic to adesired location, in accordance with an example embodiment. In thisexample embodiment, the tool graphic can be moved by performing apress-and-hold input on the graphic (shown being performed by a user'shand, and more specifically by a finger on the hand) to select thegraphic as shown in FIG. 3 b. Once the tool graphic is selected, it maygrow and include a border (e.g., a translucent border) as can be seen inFIG. 3 c. Other feedback (visual, aural, and/or tactile) may be providedto indicate that the tool graphic has been selected. Once selected, theuser can drag the tool graphic to a desired location, such as thelocation shown in FIG. 3 c in the bottom left corner of the touchscreen. The user can then release the tool graphic in the desiredlocation to reversibly lock it at that location as shown in FIG. 3 d(such that it can be moved again using the same procedure in FIGS. 3 b-3d). In some embodiments, the multi-purpose tool may be configured toallow the tool graphic to be moved in another suitable manner (e.g.,shift key plus a selection on the tool graphic and then dragging thegraphic to the desired location). In other embodiments, the tool graphicmay be permanently locked in one position (e.g., default hard-codedpositions).

FIGS. 3 e-h illustrate an example mode selection input from (or using)the multi-purpose tool graphic input to invoke a desired mode, inaccordance with an example embodiment. In this example embodiment, themulti-purpose tool graphic shown in FIG. 3 e can be tapped (orselected), as shown in FIG. 3 f, to expand the tool graphic and presentselectable mode graphics/icons/buttons as shown in FIG. 3 g. Theexpanded tool graphic shown in FIG. 3 g includes four different modebuttons, thereby providing access to invoke the modes by tapping on (orselecting) one of the buttons. The four mode buttons include a definemode button on top, a copy mode button on the bottom, a note mode buttonto the left, and a highlight mode button to the right. The expandedmulti-purpose tool graphic also includes a selectable X in the middlethat can be tapped (or selected) to turn off a currently invoked mode.FIG. 3 h illustrates a user tapping on the copy mode button located onthe bottom of the expanded multi-purpose tool graphic to invoke the copymode. An example screen shot resulting from the selection can be seen inFIG. 4 a, which will be discussed in more detail below.

FIG. 3 i shows an alternate configuration for the expanded multi-purposetool graphic, in accordance with an embodiment. As can be seen in thisconfiguration, the four buttons previously described to invoke thecorresponding four modes (define, copy, note, and highlight modes) arestill presented. However, the alternate expanded tool graphic in thisexample also includes additional note and highlight mode color buttons,which can be selected to set the initial color of the note or highlightcreated when the mode is invoked. In this example embodiment, the coloris shown for each button as a color ring surrounding the mode button (asindicated on the note mode button). Note that the color optionspresented may be hard-coded, user-configured, or some combinationthereof. For example, the colors presented may be based on the lastcolors used to create the annotations (highlights or notes) in theirrespective modes. Also note that the copy mode button is shown in whiteto indicate that this is the currently invoked mode (e.g., as selectedin FIG. 3 h). With the copy mode invoked, a user can tap (or select)away from the expanded multi-purpose tool graphic to continue operatingin the copy mode, can tap the X to turn off the copy mode (therebyresulting in mode being invoked), can select a different mode button toinvoke the corresponding mode (e.g., select the define mode button toinvoke the define mode), or perform some other suitable action (e.g.,hit a back button to return to the paginated content with the copy modeinvoked, hit the home button to exit the paginated content, etc.).

FIGS. 3 j-m illustrate example mode selection input from (or using) themulti-purpose tool graphic input to invoke a desired mode, in accordancewith an example embodiment. As can be seen, FIGS. 3 j-m illustrate thatthe tool graphic may be configured to allow a user to swipe from themulti-purpose tool graphic in the direction of the desired mode toinvoke that mode. More specifically, in this example, FIG. 3 jillustrates swiping leftward from the tool graphic to invoke the notemode, FIG. 3 k illustrates swiping rightward from the tool graphic toinvoke the highlight mode, FIG. 3 l illustrates swiping upward from thetool graphic to invoke the define mode, and FIG. 3 m illustrates swipingdownward from the tool graphic to invoke the copy mode. Note that suchswipe gestures may be performed again to turn the previously invokedmode off, or a swipe gesture in a different direction may be used toinvoke a different mode (and thereby turn the previously invoked modeoff). In this example embodiment, the mode icon associated with thedirection of the swipe gesture appears while swiping. However, the modeicon may not appear in some embodiments (e.g., the user can swipe in thedirection of a known desired mode to invoke the mode). In someembodiments, the swiping mode selection input may be used in conjunctionwith the expanded multi-purpose tool graphic configuration, such that auser can initiate a swipe gesture on the tool graphic to expand themulti-purpose tool graphic and then swipe to the presented modeicon/button to invoke the corresponding mode. As previously described,the modes may also be invoked by tapping/selecting the tool graphic toexpand the multi-purpose tool and then tapping/selecting the desiredmode to be invoked, in some embodiments. Various other suitable modeselection input may be used to invoke a mode from the multi-purpose toolgraphic and the present disclosure is not intended to be limited to theexamples provided herein.

FIGS. 4 a-d illustrate an example copy mode of an electronic computingdevice, configured in accordance with an embodiment of the presentinvention. The copy mode can be invoked using a multi-purpose toolgraphic, such as is shown in FIG. 3 h and previously described. As canbe seen in FIGS. 4 a-d, the multi-purpose tool graphic is displaying thecopy mode icon to indicate that the copy mode is invoked. In someembodiments, the user may be able to turn off the currently invoked mode(e.g., the copy mode in this example case) by tapping/selecting themulti-purpose tool graphic. In this example embodiment, the copy modeallows a user to copy selected text. FIG. 4 a shows a user tapping (orselecting) to initiate a drag, which can be used to select desired text.FIG. 4 b shows a screen shot after the user has dragged to select thedesired text section. FIG. 4 c shows a screen shot after the userreleased the drag gesture. In this example embodiment, the selected textis shown in FIG. 4 c and selected text section includes two handles oneither end of the section that can be used to expand or contract theselection in either direction (from the handle's original location) bydragging the handle as desired. Note that one of the handles is also acopy selection tool that can be tapped/selected (as opposed to beingdragged to expand/contract the selected text) to copy the selected textsection. Also note that in some embodiments, the user may be able to usethe handles to select text (to be copied) across multiple pages.Alternatively, the user may tap away from the selected text to cancelthe copy function. In another example embodiment, the selected text maybe automatically copied (e.g., to a clipboard), as soon as the userreleases the drag gesture. Various visual, aural, and/or tactilefeedback may be provided to indicate that the text was successfullycopied, such as the text copied confirmation shown in FIG. 4 d.

FIGS. 5 a-d illustrate an example define mode of an electronic computingdevice, configured in accordance with an embodiment of the presentinvention. The define mode can be invoked using a multi-purpose toolgraphic, as previously described. As can be seen in FIGS. 5 a-d, themulti-purpose tool graphic is displaying the copy mode icon to indicatethat the copy mode is invoked. In this example embodiment, the definemode allows a user to define a selected word. FIG. 5 a shows a usertapping (or selecting) to select a desired word to be defined (in thisexample case, the word is “brambles”). FIG. 5 b shows a screen shotincluding a pop-up definition box including the definition of the word.Note that the pop-up definition box has a scroll bar in this exampleembodiment to allow a user to scroll to see more of the contents of thedefinition. Also note that the definition in this example embodiment maybe pulled from any number of sources, including a dictionary providedwith the application being used to view paginated digital content (e.g.,an eReader application). In other example embodiments, the definition ofthe selected word may be presented in another suitable manner, such asopening a web browser or external application to provide a definition ofthe word. Continuing with the example embodiment shown in FIG. 5 b, thepop-up box may be closed by tapping outside of the box to close it, suchas is shown in FIG. 5 c. In some embodiments, the user can then selectanother word from the page displayed in FIG. 5 d or navigate to anotherpage and select a word to be defined. However, in other embodiments, thecurrently displayed/presented page (e.g., page 10 in this example case)may be locked such that the user cannot navigate away from the pageuntil the define mode is turned off.

In some embodiments, a lookup mode may be invoked using themulti-purpose tool. When invoked, the lookup mode may allow a user toselect a word to cause the definition of that word to be presented,similar to the functionality of the define mode. However, the lookupmode may also allow the user to obtain other information about selectedwords, phrases, or images, for example. In one or more embodiments, thelookup mode may allow a user to select one or more words to lookup thosewords using an internet search engine (e.g., Google® or Bing® searchengines), using a language translation service (e.g., to translateselected English words to Spanish), or using an encyclopedia (e.g.,Wikipedia® encyclopedia), for example. In another example, in oneembodiment, the lookup mode may allow a user to select an image tolookup that image using an image recognition service (e.g., Google®search engine's search by image feature), for example. Numerous otherlookup services may be used in conjunction with the lookup mode and, insome embodiments, the results of selecting a word, phrase, or image tolookup related information may be presented within the same applicationbeing used to present/display the paginated digital content.

FIGS. 6 a-d illustrate an example highlight mode of an electroniccomputing device, configured in accordance with an embodiment of thepresent invention. The highlight mode can be invoked using amulti-purpose tool graphic, as previously described. As can be seen inFIGS. 6 a-d, the multi-purpose tool graphic is displaying the highlightmode icon to indicate that the highlight mode is invoked. In thisexample embodiment, the highlight mode allows a user to highlight text.FIG. 6 a shows a user tapping (or selecting) the paginated content toinitiate a drag, which can be used to select text to be highlighted. Insome embodiments, the tap (or selection) to initiate the drag has to beperformed over a word or within the text to create a highlight. FIG. 6 bshows a screen shot after the user has dragged to highlight the desiredtext section. FIG. 6 c shows a screen shot after the user released thedrag gesture to complete the highlighting. In some embodiments, draggingto highlight may highlight entire words at a time, while in otherembodiments, dragging to highlight may highlight individual charactersat a time. In yet another embodiment, individual characters may behighlighted until a space within the text is highlighted, causing onlyentire words to be highlighted after the space has been highlighted.

In this example embodiment, the highlighted text section isautomatically selected after the highlighting is created to allow thehighlighted section to be edited. When selected, the highlighted textsection includes two handles on either end of the section, as can beseen in FIG. 6 c, that can be used to expand or contract thehighlighting in either direction (from the handle's original location)by dragging the handle as desired. Note that in this example embodiment,one of the handles is also a color picker tool, which can be used tochange the highlighting color or to delete the highlighting (byselecting the X deletion graphic on the current highlight color in thisexample case), in addition to being able to expand/contract thehighlighting. Also note that in some embodiments, the user may be ableto use the handles to expand the highlighting across multiple pages. Insome embodiments, when dragging a handle to expand/contract thehighlighting, the handle being dragged, upon release, expands to becomethe color picker tool. As can be seen in FIG. 6 d, the user can tap (ormake a selection) away from the highlight to deselect the highlight,thereby causing the highlight to no longer be editable and causing thehandles (including the color picker tool) to stop beingpresented/displayed. Note that in some embodiments, a user can re-selecta highlight (e.g., to edit the highlight) by tapping on it. In some suchembodiments, the highlight mode (or note mode or annotation mode) mayneed to be invoked before re-selecting the highlight.

FIGS. 7 a-d, 8 a-h, and 9 a-d illustrate an example note mode of anelectronic computing device, configured in accordance with one or moreembodiments of the present invention. As will be apparent in light ofthis disclosure, the note mode may allow for the creation and editing ofmultiple note types including (but not limited to) sticky notes, marginnotes, and/or highlight notes. In accordance with one or moreembodiments, FIGS. 7 a-d relate to the creation and editing of a stickynote, FIGS. 8 a-h relate to the creation and editing of a margin note,and FIGS. 9 a-d relate to the creation and editing of a highlight note.Remember that the note mode may also be referred to herein and in thefigures as an annotation mode (since the annotation mode may include allof the functionality of the note mode, in some embodiments). The notemode can be invoked using a multi-purpose tool graphic, as previouslydescribed. As can be seen in FIGS. 7 a-d, 8 a-h, and 9 a-d, themulti-purpose tool graphic is displaying the note mode icon to indicatethat the note mode is invoked.

FIGS. 7 a-d illustrate creating and editing a sticky note using anexample note mode of an electronic computing device, in accordance withan embodiment of the present invention. FIG. 7 a shows a user tapping(or selecting) the paginated content to create a sticky note at thelocation where the user tapped (or selected). FIG. 7 b shows a screenshot after the sticky note has been created. As can be seen, a stickynote edit box is automatically presented, allowing the user to entertext at the cursor location using, for example, the virtual keyboardthat is also presented in this example embodiment. Note that the stickynote edit box in this example embodiment includes a color picker toolthat can be used to move the edit box (e.g., by dragging the colorpicker tool), to change the color of the sticky note (e.g., by tappingon/selecting the desired color circle), to delete the sticky note (e.g.,by tapping on/selecting the X graphic on the currently selected color),or to toggle the note to a margin note using the note toggle tool (e.g.,by selecting the margin note icon, as will be described in more detailbelow). In some embodiments, the user may be able to edit or configurethe font (size, color, type, etc.), alignment (left, center, right,etc.), and or other properties of the text entered for a sticky note,margin note, and/or highlight note.

FIG. 7 c shows a screen shot after the user has entered text (“Samplenote”) into the edit box (e.g., using the virtual keyboard). As can alsobe seen, the user has tapped (or selected) outside of the edit box tosave the sticky note as shown in FIG. 7 d. In some embodiments, acreated/edited note may be saved from the edit box (for either a stickynote, margin note, or highlight note) using some other action, such ashitting a save button that is present in the note edit box in anotherembodiment, or some other suitable action as will be apparent in lightof this disclosure. In some embodiments, tapping outside of the edit box(for either a sticky note, margin note, or highlight note) prior toentering text and/or when the edit box is empty, may cause the note tonot be saved. Note that the sticky note saved in FIG. 7 d is in thelocation of the initial tap to create sticky note location of FIG. 7 a.Also note that the sticky note and associated edit box have the samecolor. In some embodiments, the sticky note can be further edited bytapping on (or selecting) the note to cause the associated sticky noteedit box to be presented again. In some embodiments, the sticky note canbe moved by press-and-holding on the sticky note (when not in edit mode)to grab it, dragging the grabbed sticky note to a desired location, andthen dropping the sticky note at the desired location (e.g., as will bediscussed with reference to FIG. 9 b).

FIGS. 8 a-h illustrate creating and editing a margin note using anexample note mode of an electronic computing device, in accordance withone or more embodiments of the present invention. In this exampleembodiment, margin notes can be created by converting a previouslycreated sticky note to a margin note. For ease of description, thesticky note created in FIGS. 7 a-d will be used to illustrate creatingthe margin note in this example embodiment. FIG. 8 a shows thepreviously created sticky note in edit mode (with the edit boxpresented, e.g., as shown in FIG. 7 c). As can also be seen, the user istapping (or selecting) the margin note icon in the note toggle tool toconvert the note type from a sticky note to a margin note. FIG. 8 b is ascreen shot showing the result of the conversion. As can be seen, thecontents of the sticky note have been converted to a margin note,allowing the margin note to be edited in a manner similar to how asticky note can be edited (e.g., editing the text using the virtualkeypad at the bottom of the screen). Note that the contents of the note(“Sample note”) were maintained when the note was converted from asticky note to a margin note. In this example embodiment, the marginnote includes a color picker (similar to the color picker shown for thesticky note); however, a garbage can deletion icon is now present on thecurrent color of the margin note and the garbage can icon can beselected to delete the note. A selectable X graphic is also included,which can be selected to save the note and close out from editing thecontents of the margin note. FIG. 8 c shows the user tapping (orselecting) the selectable X graphic to save all of the edits made to themargin note (e.g., the color of the note was changed as can be seenbetween FIGS. 8 b and 8 c by the change in location of the garbage candeletion icon). In some embodiments, sticky notes may have a similarcolor picker as the margin note shown in FIG. 8 b (e.g., including thegarbage can deletion icon on the current color of the note andselectable X graphic to save the note), while in some embodiments,margin notes may have a similar color picker as the sticky note shown inFIG. 8 a (e.g., including an X deletion icon on the current color of thenote and an edit box that a user can select outside of to save thenote).

FIG. 8 d is a screen shot showing the result of tapping the X graphic tosave the margin note as shown in FIG. 8 c. As can be seen in FIG. 8 d,after the user tapped the X graphic to save the margin note contents,the margin note is presented over the text of the paginated digitalcontent, inside of a bounding box having ratio handles. Note that themargin note location is the same as the sticky note location from thepreviously created sticky note (as shown in FIG. 7 d), in this examplecase. The bounding box allows the user to rotate, resize, change theratio of, and move the margin note, each of which will be discussed inturn. FIG. 8 e illustrates the bounding box being rotated to rotate themargin note. As can be seen, the user used a two finger twist gesture onthe bounding box to rotate the box and the margin note counterclockwiseninety degrees. In some embodiments, the bounding box may have specificrotation stop points (e.g., every forty-five or ninety degrees fromhorizontal), while in other embodiments, the bounding box may be able tobe freely rotated to any angle. In yet other embodiments, the boundingbox may have stop points when rotated in one direction (e.g.,counterclockwise) and be capable of being freely rotated in the otherdirection (e.g., clockwise), for example. Between FIGS. 8 e and 8 f, theuser may have used a two finger pinch-in gesture over the margin note(not shown) to resize/scale, and more specifically, shrink, the marginnote. In this example embodiment, a two-fingered pinch-in gestureperformed on the note can be used to shrink the margin notetext/contents, and a two-fingered spread-out gesture performed on thenote can be used to enlarge the note text/contents. Non touch-sensitiveinput may be used to perform the two-fingered gestures used to edit themargin note while in the bounding box as described herein, as will beapparent in light of this disclosure.

FIG. 8 f illustrates the bounding box being resized using the ratiohandles to reflow/scale the margin note to best fit within the boundingbox (e.g., to avoid hyphenation, if possible). As can be seen, the tophandle was dragged down to decrease the size of the box and the marginnote contents adjusted accordingly. If the two-fingered pinch-in gesturewas previously performed to shrink the margin note text, then thegesture in FIG. 8 f may have only caused a reflow of the margin notetext as shown. If the two-fingered pinch-in gesture was not previouslyperformed, then the gesture in FIG. 8 f may have caused both a reflowand a scaling (scale down, in this example case) of the margin note textas shown. In this example embodiment, any four of the ratio handles maybe used to resize the bounding box. In other example embodiments, thebounding box may have fewer or additional handles to allow resizing themargin note. For example, the bounding box may have only two handles(e.g., one in each corner). In such an example, the two bounding boxhandles may allow for resizing and/or reflowing of the content (e.g.,when one or both handles are dragged to expand or contract the box andits contents) and for rotating the box (e.g., by hovering over one ofthe handles to bring up a rotation wheel that can be rotated to rotatethe bounding box and its contents). Numerous other configurations andmargin note bounding box variations may be apparent in light of thisdisclosure.

FIG. 8 g illustrates the bounding box being moved to move the marginnote. As can be seen, the user performed a press-and-hold gesture on thebounding box and then dragged the box to a desired location, and thenreleased the box to move it to the desired location shown. In someembodiments, margin notes can be moved anywhere on the paginated digitalcontent, including over highlighted text. FIG. 8 h shows the usertapping (or selecting) outside of the bounding box to save the marginnote and exit the bounding box editing. Note that in this exampleembodiment, the margin note contents are immediately visible (comparedto, e.g., the sticky note, which has to be selected to view itscontents). In some embodiments, the user may be able to enter boundingbox editing by tapping (or selecting) once on the margin note and enterthe edit box editing by tapping (or selecting) twice on the margin note(or once when the margin note is in bounding box editing). Therefore,the margin note has two levels of editing that can be accessed. Notethat when the margin note is in edit box editing (e.g., as shown in FIG.8 b), the user can use the note toggle tool to convert a margin note toa sticky note. In some embodiments, a margin note can only be createdfrom a sticky note.

FIGS. 9 a-d illustrate creating and editing a highlight note using anexample note mode of an electronic computing device, in accordance withan embodiment of the present invention. In this example embodiment,highlight notes can be created by combining a sticky note with ahighlight. For ease of description, the sticky note created in FIGS. 7a-d and the highlight created in FIGS. 6 a-d will be used to illustratecreating the highlight note in this example embodiment. FIG. 9 a showsthe previously created sticky note and highlight. As can be seen in FIG.9 b, the user has performed a press-and-hold to grab the sticky note andthen dragged the grabbed sticky note over the highlighted text. In thisexample embodiment, the sticky note grows and includes a border (e.g., atranslucent border) when grabbed, as shown in FIG. 9 c. As can also beseen, when the grabbed highlight appears over the highlighted text, thehighlight appears selected as shown. Other feedback (visual, aural,and/or tactile) may be provided to indicate that the sticky note hasbeen grabbed, that the grabbed sticky note is hovering over thehighlight and/or that dropping the sticky note will create a highlightnote. For example, the highlight note as shown in FIG. 9 d may start toappear to indicate a highlight note will be formed, or some othersuitable feedback.

FIG. 9 c is a screen shot showing the highlight note created after thesticky note was released on the highlight in this example embodiment. Ascan be seen, a highlight note edit box is automatically presented,allowing the user to enter text at the cursor location using, forexample, the virtual keyboard that is also presented in this exampleembodiment. The highlight note edit box in this example embodimentallows the highlight note to be edited in a manner similar to how asticky note can be edited (using its corresponding edit box), aspreviously described. However, the highlight note edit box does notinclude a note toggle tool, since a highlight note cannot be convertedto a margin note, in this example embodiment. As can also be seen, theuser in this example embodiment entered text (“Sample highlight note”)into the edit box. Note that in this example embodiment, the highlightnote edit box includes a tail showing the highlight that is attached to.Also note that the highlight note maintained its own color (the colorfrom the sticky note) and did not inherit the color of the highlight itwas combined with.

FIG. 9 d is a screen shot showing the highlight note saved after theuser has tapped (or selected) outside of the highlight note edit box tosave the highlight note. As can be seen, the highlight note is locatedat the beginning of the highlight in this example embodiment and thehighlight note color is independent of the associated highlight color.In some embodiments, the created highlight note can be changed back to asticky note by dragging the highlight note away from its associatedhighlight. In some embodiments, the user may have to press-and-hold onthe highlight note before being able to drag it away from a highlight tochange it to a sticky note. Remember that annotations as used herein mayinclude highlights and/or notes added to paginated digital content, andthat the functionality of the highlight and note modes (as variouslydescribed herein) may be collectively included in an annotation mode.For example, in some embodiments, the annotation mode (when invoked) maybe configured to allow a user a to drag over text to highlight it andtap to create sticky notes, as well as select previously createdhighlights and notes to edit and interact with them (e.g., to create ahighlight note as described in FIGS. 9 a-d).

As will be apparent in light of this disclosure, the annotations asvariously described herein (e.g., highlights, sticky notes, marginnotes, and highlight notes) may have an affinity to text or locationswithin the paginated digital content. For example, a highlight isassociated with or has an affinity to the text it is associated with andsticky notes and margin notes have an affinity to the locations withinthe paginated content where they were created or moved to. Therefore,when the paginated digital content is modified to change, for example,the viewing preferences of the paginated content (e.g., zoom in/out,increase/decrease font size or margins, etc.), the annotations may movewith the paginated content to maintain their position relative to thetext (or other content) of the paginated content they were originallyassociated with. For example, if the eBook viewing preferences for theeBook shown in FIG. 6 d were changed to make the text larger, thehighlight associated with “the brambles . . . green mounds.” would stillbe associated with that textual passage and thus become larger with thetext to ensure that the text is still highlighted in the manner shown.

Merging, Deleting, Filtering, and Searching Annotations

FIGS. 10 a-d and 11 a-h illustrate example merge annotation functionsfor an annotation mode of an electronic computing device, in accordancewith one or more embodiments of the present invention. Morespecifically, FIGS. 10 a-d illustrate merging a previously createdsticky note with another previously created sticky note and FIGS. 11 a-hillustrate merging a previously created highlight with anotherpreviously created highlight (where each highlight has its own highlightnote). FIGS. 12 a-h illustrate example delete annotation and undo deleteannotation functions for an annotation mode of an electronic computingdevice, in accordance with one or more embodiments of the presentinvention. Note that although the undo delete annotation function shownin FIGS. 12 e-h is in the context of undoing one or more annotationdeletions, the undo feature may also be used for undoing the merging ofannotations, in some embodiments. The annotation mode can be invokedusing a multi-purpose tool graphic, as previously described. In someembodiments, the annotation mode may be invoked upon the selection orgrabbing of a previously created annotation. As can be seen in FIGS. 10a-d, 11 a-h, and 12 a-h, the multi-purpose tool graphic is displaying anannotation mode icon to indicate that the annotation mode is invoked.These examples are provided for illustrative purposes and are notintended to be limiting. Numerous other variations of merging and/ordeleting annotations (e.g., highlights and/or notes added to paginateddigital content), as well as undoing such merges and/or deletions, willbe apparent in light of the present disclosure.

FIGS. 10 a-d illustrate merging sticky notes added to paginated digitalcontent, in accordance with an embodiment of the present invention. FIG.10 a shows two previously created sticky notes: sticky note 1 havingcolor 1 and sticky note 2 having color 2. As can be seen, the user hasperformed a press-and-hold on sticky note 1 to grab it. In this exampleembodiment, the sticky note grows and includes a border (e.g., atranslucent border) when grabbed, as shown. FIG. 10 b shows the userdragging the grabbed sticky note 1 and dropping it on sticky note 2. Inthis example embodiment, when the grabbed sticky note (e.g., sticky note1 in this example case) is hovering over the sticky note to be droppedon (e.g., sticky note 2 in this example case) the sticky note to bedropped on grows and includes a border (e.g., a translucent border),similar to the grabbed sticky note, as shown in FIG. 10 b. Otherfeedback (visual, aural, and/or tactile) may be provided to indicatethat the sticky note has been grabbed, that the grabbed sticky note ishovering over another sticky note and/or that dropping the sticky notewill merge the sticky notes.

FIG. 10 c is a screen shot showing the merged sticky note edit box aftersticky note 1 was released on sticky note 2 in this example embodiment.As can be seen, the merged sticky note inherited the color of stickynote 2 (color 2). As can also be seen, the contents of sticky note 1have been added to the end of the contents of sticky note 2. In someembodiments, a blank line or other suitable separator, may be enteredbetween the contents of the merged notes. After tapping outside of themerged sticky note edit box, as shown in FIG. 10 c, the merged stickynote can be saved in the location shown in FIG. 10 d. As can be seen,the location of sticky note 2 (and the color, as previously described)were inherited in this example embodiment. In other embodiments, thesticky note that was dragged and dropped (e.g., sticky note 1, in theprevious example case) may dictate the color, location, and/or initialcontent of the merged note.

Although sticky notes are being merged together in the merging exampleshown in FIGS. 10 a-d and described above, sticky notes can also bemerged with highlight notes in a similar manner. However, one differencefrom the merging example shown in FIGS. 10 a-d may be that, in someembodiments, when merging a sticky note with a highlight note, thegrabbed sticky note can be dragged and dropped onto any portion of ahighlight already containing a highlight note to combine the grabbedsticky with the highlight note. Note (as previously described) that oncea highlight note is grabbed (e.g., using a press-and-hold input) anddragged away from its associated highlight, it may become a sticky note.Therefore, if a user wants to grab a highlight note and drag it awayfrom its associated highlight to merge it with a sticky note or anotherhighlight note, such actions can be tantamount to merging a sticky notewith the sticky note or other highlight note.

FIGS. 11 a-h illustrate merging highlights including highlight notesadded to paginated digital content, in accordance with an embodiment ofthe present invention. FIG. 11 a shows two previously created highlights(each including associated highlight notes): highlight 1 having color 1(and an associated highlight note 1 having color 2) and highlight 2having color 3 (and an associated highlight note 2 having color 4). FIG.11 b is a screen shot showing highlight 2 selected in response to a usertapping on (or selecting) highlight 2. Note that the handles, includingthe color picker tool appear to indicate that highlight 2 is selected.FIG. 11 c illustrates a user dragging the top handle of highlight 2 todrag highlight 2 to highlight 1, such that highlight 2 temporarilyoverlaps highlight 1 as shown. Note in this example embodiment that theoverlap need not be a complete overlap, but it could be (e.g., where ahighlight is totally encompassed by the highlight beingdragged/expanded). In some embodiments, only a character may have tooverlap when dragging to merge highlights. Also note that highlight note2 follows along with highlight 2 in this example embodiment while it isbeing dragged using the handle as shown in FIG. 11 c. In someembodiments, feedback may be provided to indicate that highlight 2 hasbeen temporarily dragged to (and overlaps with) highlight 1. Suchfeedback may include visual, aural, and/or tactile feedback to indicate,for example, that dropping the dragged handle of highlight 2 would causehighlight 1 and highlight 2 to merge.

FIG. 11 d is a screen shot showing the merged highlights (including themerged highlight notes) after the user released the dragged highlight 2at the point shown in FIG. 11 c to merge the highlights together. As canbe seen, the merged highlight inherited the color of highlight 2 (color3) and the merged highlight extends to the beginning of highlight 1. Ascan also be seen, the merged highlight note inherited the color ofhighlight note 2 (color 4) and is located at the beginning of the mergedhighlight. Therefore, the color of the dragged highlight and highlightnote were inherited in this example embodiment. In other embodiments,the highlight dragged to (e.g., highlight 1, in the previous examplecase) may dictate the color of the resulting merged highlight. Also, inother embodiments, the highlight note of a highlight dragged to (e.g.,highlight note 1, in the previous example case) may dictate the color ofthe resulting merged highlight note and/or dictate the order of thecontent in the merged highlight note. In some embodiments, where onlyone of the highlights being merged has an associated highlight note, themerged highlight note essentially remains the same (except that will belocated in front of the merged highlight).

In this example embodiment, and as shown in FIG. 11 d, the mergedhighlight is selected after the highlight merging has been performed.FIG. 11 e shows the user tapping (or selecting) away from the highlightto deselect the highlight. FIG. 11 f shows the user tapping (orselecting) the merged highlight note to select it and cause the mergedhighlight note edit box shown in FIG. 11 g to appear. Note that in someembodiments, when merging highlights where at least one of the mergedhighlights has an associated highlight note, the annotation mode may beconfigured to present/display the merged highlight note edit box. Forexample, in such a configuration, the merged highlight note edit boxshown in FIG. 11 g would have presented itself in FIG. 11 d, when theuser released dragging highlight 2 to merge highlight 2 and highlight 1.FIG. 11 g shows that the contents of highlight note 1 have been added tothe end of the contents of highlight note 2. In some embodiments, ablank line or other suitable separator, may be entered between thecontents of the merged notes. FIG. 11 g also shows a user tapping (orselecting) the X graphic on the current color of the merged highlightnote to delete the merged highlight note. FIG. 11 h shows the mergedhighlight after the merged highlight note has been deleted.

In some cases, a highlight may be expanded by dragging one of itshandles, and during such an expansion, the highlight may overlap orencompass one or more highlights. In such cases, the highlight beingdragged/modified, in some embodiments, may dictate the color of thefinal merged highlight. In such cases, if the highlight beingdragged/modified encompasses other highlights having highlight notes,the color and content of the resulting merged highlight note associatedwith the resulting merged highlight may be dictated based on one of twoscenarios. In a first scenario, where the highlight being dragged has ahighlight note, the color of the resulting merged highlight note willbe, in an example embodiment, the color of the dragged highlight note,and the content of the merged highlight note will be dictated by theorder in which the highlight notes were overlapped (with the content ofthe highlight note of the highlight being dragged being on top). In asecond scenario, where the highlight being dragged does not have ahighlight note, the color of the resulting merged highlight note willbe, in an example embodiment, the color of the first highlight noteoverlapped, and its content will be dictated by the order in which thehighlight notes were overlapped. In some cases a highlight may beexpanded by dragging one of its handles, and during such an expansion,the highlight may overlap or encompass one or more sticky notes. In suchcases, the scenarios laid out for overlapping/encompassing highlightnotes may also apply to overlapping/encompassing sticky notes, inaccordance with some embodiments. The techniques illustrated in FIGS. 11a-h and described herein can also be used to merge a previously createdhighlight with a new highlight being created. For example, if you tapand drag to create a new highlight having a blue color and continue thedrag over an existing highlight that is yellow, the existing highlightcan be merged into the new highlight when the drag is released,resulting in a merged highlight that is blue.

FIGS. 12 a-d illustrate deleting annotations added to paginated digitalcontent, in accordance with one or more embodiments of the presentinvention. FIG. 12 a shows a previously created sticky note andpreviously highlighted text. As can be seen, a user has performed apress-and-hold on the sticky note to grab it. In this exampleembodiment, the grabbed sticky note grows and includes a border (e.g., atranslucent border) when grabbed, as shown in FIG. 12 a. FIG. 12 b showsthe user flicking the grabbed sticky note to delete it. In someembodiments, the sticky note may animate off the screen when it isflicked to delete it. FIG. 12 b shows, in this example embodiment, thesticky note fading off the screen in the direction it has been flicked.Other feedback (visual, aural, and/or tactile) may be provided toindicate that a sticky note has been grabbed or to indicate that asticky note is being deleted or has been deleted. In this exampleembodiment, the flick input to delete the sticky note may be in anydirection after grabbing the sticky note. In some embodiments, thepress-and-hold (to grab) then flick to delete input as variouslydescribed herein may be used to delete margin notes and/or highlightnotes.

FIG. 12 c shows that the user has performed a press-and-hold on thesticky note to grab it. In this example embodiment, the highlightcollapses into an upside down teardrop (with instructions inside stating“Flick to delete”) when grabbed, as shown in FIG. 12 c. In someembodiments, the annotation mode may be configured to not collapsegrabbed highlights or to collapse a grabbed highlight into anothersuitable representation, such as a circle, a ball, a teardrop shape, ora crumpled up piece of paper, for example. FIG. 12 d shows the userflicking the grabbed (and collapsed) highlight to delete it. In someembodiments, the grabbed highlight may animate off the screen when it isflicked to delete it. FIG. 12 d shows, in this example embodiment, thegrabbed highlight (which collapsed into an upside down teardrop in thisexample case) fading off the screen in the direction it has beenflicked. Other feedback (visual, aural, and/or tactile) may be providedto indicate that a highlight has been grabbed or to indicate that ahighlight is being deleted or has been deleted. In this exampleembodiment, the flick input to delete the highlight may be in anydirection after grabbing the highlight. In some embodiments, a highlightnote associated with a highlight being deleted using a press-and-hold(to grab) then flick input may be deleted as well. In other embodiments,the highlight note associated with the highlight being deleted maydetach from the highlight and change to a sticky note after thehighlight is deleted.

In some embodiments, annotations may be deleted using other suitabletechniques or based on other suitable deletion input. For example, andas previously described, annotations (highlights and/or notes) may bedeleted by selecting an X or garbage can deletion graphic associatedwith the annotation. Such a selectable deletion graphic may be presentedwhen editing the annotation. For example, the X or garbage can deletiongraphic may be located on a color picker tool presented when editing ahighlight or note (e.g., as shown in FIG. 6 c for a highlight, FIG. 7 cfor a sticky note, FIG. 8 c for a margin note, and FIG. 9 c for ahighlight note). A user can tap on or select the X or garbage candeletion graphic when presented to delete the annotation (e.g., as shownin FIGS. 11 g-h). Various other suitable deletion input will be apparentin light of this disclosure.

FIGS. 12 e-h illustrate undoing the annotation deletions performed inFIGS. 12 a-d, in accordance with one or more embodiments of the presentinvention. FIG. 12 e shows a user shaking (or providing shake input to)the electronic computing device to undo the last deletion performed. Ascan be seen in FIG. 12 f, after the deletion was undone in response tothe shake input, the deleted highlight returned to the text from thepaginated digital content it was associated with prior to deletion(e.g., compare FIG. 12 a to FIG. 12 f). In some embodiments, ananimation may be provided to indicate that the deletion has been undone.For example, if a user performs a press-and-hold then flick input toflick the annotation off the screen in the direction of the flick andthereby delete the annotation (e.g., as shown in FIGS. 12 a-b and inFIGS. 12 c-d), undoing the deletion may cause the annotation to animateback to its location on the paginated digital content in a directionopposite the way it was flicked off the screen. In another example, if auser selects a deletion graphic (e.g., an X or garbage can) associatedwith an annotation to delete it (e.g., as shown in FIGS. 11 g-h),undoing the deletion may cause the annotation to animate back to itslocation on the paginated digital content by fading in on that location.FIG. 12 g shows a user performing a Z-shaped gesture to undo the lastdeletion performed. In this example case, since the deletion of thehighlight was undone, the last deletion performed prior to thatdeletion, was the deletion of the sticky note as shown in FIG. 12 b. Ascan be seen in FIG. 12 h, after the deletion was undone in response tothe Z-shaped gesture input, the deleted sticky note returned to thepaginated digital content at the spot it was located prior to deletion(e.g., compare FIG. 12 a to FIG. 12 h).

In some embodiments, merging annotations (as variously described herein)may be undone in a similar manner to undoing annotation deletions.Undoing the merging of two (or more) annotations, in some embodiments,may cause merged annotations to return to their previous state prior tomerging in response to an undo input. Therefore, in some embodiments,the annotations mode may be configured to undo the most recentannotation deletion and/or merge in response to an undo input. Note thatthe shake and Z-shaped gesture undo input shown in FIGS. 12 e and 12 gare provided as examples for illustrative purposes and are not intendedto limit the present disclosure. Various other suitable undo input maybe used to undo one or more annotation deletions and/or merges. Forexample, in some embodiments, undo input may include a Ctrl+Z command(using a keyboard), selecting an undo button from a dialog (which maypop up in response to deleting or merging an annotation), or some othersuitable input as will be apparent in light of this disclosure.

FIGS. 13 a-c illustrate example filtering and searching functions for anannotation mode of an electronic computing device, in accordance withone or more embodiments of the present invention. The screen shots inFIGS. 13 a-c each include a search bar at the top left of the screen anda filter selection section at the top middle of the screen. FIG. 13 a isa screen shot showing only notes for paginated digital content, asindicated the show notes filter selection. As can be seen, the resultingdisplay includes sticky notes, margin notes, and highlight notes. Thehighlight notes also include the associated highlights below each note,initially in a collapsed preview view, to, for example, provide contextfor the highlight note. FIG. 13 b is a screen shot showing the expandedview of a highlight associated with a highlight note, as can be seen inthe white section on the right side of the screen. The expanded view wasdisplayed in this example embodiment as a result of selecting the downarrow next to the collapsed preview of the associated highlight (causingthe arrow to point upward when the view is expanded). FIG. 13 c is ascreen shot showing the filtered search results when searching for theterm “carbon.” As can be seen, only notes including the “carbon” termare displayed, with the search term in bold in each note foundcontaining the term (e.g., to identify where in the notes the termappears).

As will be apparent in light of this disclosure, in some embodiments,the filtering and searching functionality previously described may beused with highlights to, for example, filter them out and only viewhighlighted passages, or to search them and only find search resultsfrom within the highlighted content. In the screen shots shown in FIGS.13 a-c, the highlights filter icon is located to the left of the notesfilter icon. In some embodiments, the filtering and searchingfunctionality previously described may be used with all annotations,including notes and highlights. In such embodiments, a user may be ableto view all annotations at once (e.g., in the list view shown in FIGS.13 a-c) or search for a term or phrase within all annotations. In someembodiments, the filtering/searching functionality may allow a user tofilter out and/or search specific note types (e.g., sticky notes, marginnotes, and/or highlight notes). In some embodiments, thefiltering/searching functionality may allow a user to filter out and/orsearch all annotations (e.g., highlighted content, notes, and/orbookmarks or flags) of a certain color. Numerous variations andconfigurations will be apparent in light of this disclosure.

Methodology

FIG. 14 illustrates a method for providing a multi-purpose tool in anelectronic computing device, in accordance with one or more embodimentsof the present invention. FIG. 15 illustrates a method for providing anannotation mode in an electronic computing device, in accordance withone or more embodiments of the present invention. The examplemethodologies may be implemented, for instance, by the UI module of theexample electronic computing device shown in FIG. 2 a, or the exampleelectronic computing device shown in FIG. 2 b (e.g., with the UIprovisioned to the client by the server). To this end, the UI may beimplemented in software, hardware, firmware, or any suitable combinationthereof, as will be appreciated in light of this disclosure. Incomputing devices including a touch screen or other touch-sensitiveinterface, the computing device may be able to detect contact, whetherdirect or proximate (e.g., via hovering input), which can be used invarious ways to implement UI functionality, including performing a modeselection input using the multi-purpose tool, a deletion input to deletean annotation (e.g., a highlight and/or a note), or other input asvariously described herein.

In the example case illustrated in FIG. 14 the method starts bydetermining 101 if paginated digital content (or similarly divideddigital content) is being presented/displayed. The paginated digitalcontent may be presented/displayed using any appropriate application orprogram, such as an eReader application, an office application (e.g.,document viewing application), a magazine/catalog reader application, orany other suitable application or program. If paginated digital contentis not being displayed (or otherwise presented), the method continues toreview until such content is displayed. Once paginated digital contentis being displayed, the method continues by determining 102 if amulti-purpose tool (as variously described herein) is enabled. In someembodiments, the multi-purpose tool may be enabled from aconfiguration/settings menu or it may be enabled when a specificapplication or program is running, for example. In some embodiments, themulti-purpose tool may be tied to one or more applications capable ofpresenting/displaying paginated digital content and the tool may not beavailable/enabled until one such application is running, active, and/ordisplayed (which may result in a conservation of device power and/ormemory).

If the multi-purpose tool is not enabled, then the method continues byreviewing 103 if the paginated digital content is still being displayed.If the content is not still being displayed, the method continues backat diamond 101 to review if paginated digital content is beingdisplayed. If the paginated digital content is still being displayed,the method continues to review until either the multi-purpose tool isenabled (e.g., enabled from the application presenting/displaying thepaginated digital content) or until the paginated digital content is nolonger being displayed. Remember, paginated digital content as usedherein includes content that is divided in an organized manner. Forexample, a gallery application configured to present/display discretephotographs may be considered paginated digital content and may benefitfrom the multi-purpose tool as variously described herein.

Once the multi-purpose tool has been enabled, the method continues bydetermining 104 if a mode selection input has been performed to invoke adesired mode. Mode selection input may include swiping from themulti-purpose tool in the direction of a desired mode,tapping/press-and-holding the multi-purpose tool and then swiping to thedesired mode while maintaining contact, tapping the multi-purpose tool(e.g., to expand the tool to show the available modes) and then tappingthe desired mode, or any other suitable input as will be apparent inlight of this disclosure. FIGS. 3 e-m show example mode selection input,in accordance with some embodiments. Continuing from diamond 104 of FIG.14, if a mode selection input has not been performed, the methodcontinues by reviewing 103 if the paginated digital content is stillbeing displayed (and also, in some cases, if 102 the multi-purpose toolis still enabled). If a mode selection input has been performed toinvoke a desired more, the method continues by invoking 105 the desiredmode. Example modes may include an annotation mode, highlight mode, notemode, copy mode, define mode, or lookup mode, all of which are variouslydescribed herein, or some other suitable mode as will be apparent inlight of this disclosure.

After the desired mode is invoked, the method continues by determining106 if user input has been performed on the paginated digital content.User input may include dragging over text of the paginated content(e.g., to create a highlight) or tapping/selecting the paginated content(e.g., to create a sticky note), as previously described. If user inputhas not been performed, the method continues by determining 108 if themode is still invoked. If user input has been performed, the methodcontinues by performing 107 a function determined by the invoked modeand user input. For example if the user input is dragging over text ofthe paginated content, the function performed would depend on theinvoked mode. In such an example, if a highlight or annotation mode (asvariously described herein) is the invoked mode, then the functionperformed may be to highlight the text that was dragged over. However,if the invoked mode is a copy mode (as variously described herein), thenthe function performed may be to copy the text that was dragged over (orto select the text and request copy confirmation, for example).

After the function determined by the invoked mode and user input isperformed 107, the method continues by determining 108 if the mode isstill invoked. If the mode is still invoked, the method continues toreview if 106 user input has been performed on the paginated digitalcontent until either such user input has been performed/received oruntil the mode is no longer invoked. In some embodiments, the mode maybe turned off or no longer be invoked if another mode isselected/invoked, if the multi-purpose tool is disabled, if thepaginated digital content is closed, and/or if a home or power button isselected, just to list a few possible examples. If the mode is not stillinvoked, the method continues to diamond 103 to determine if thepaginated digital content is still being presented/displayed.Alternatively, the method may continue to diamond 102 (to determine ifthe multi-purpose tool is still enabled) or to diamond 104 (to determineif a mode selection input has been performed to invoke a desired mode).

In the example case illustrated in FIG. 15 the method starts bydetermining 201 if paginated digital content (or similarly divideddigital content) is being presented/displayed. As previously described,the paginated digital content may be presented/displayed using anyappropriate application or program, such as an eReader application, anoffice application (e.g., document viewing application), amagazine/catalog reader application, or any other suitable applicationor program. If paginated digital content is not being displayed (orotherwise presented), the method continues to review until such contentis displayed. Once paginated digital content is being displayed, themethod continues by determining 202 if an annotation mode (as variouslydescribed herein) is invoked. As previously described, in someembodiments, the functionality of the annotation mode may be included intwo separate modes, such as a highlight mode and a note mode. In someembodiments, the annotation mode may be invoked (and/or enabled) from aconfiguration/settings menu or it may be invoked when a specificapplication or program is running, for example. In some embodiments, theannotation mode may be tied to one or more applications capable ofpresenting/displaying paginated digital content and the mode may not beavailable/enabled/invoked until one such application is running, active,and/or displayed (which may result in a conservation of device powerand/or memory).

If the annotation mode is not invoked, then the method continues byreviewing 203 if the paginated digital content is still being displayed.If the content is not still being displayed, the method continues backat diamond 201 to review if paginated digital content is beingdisplayed. If the paginated digital content is still being displayed,the method continues to review until either the annotation mode isinvoked (e.g., invoked from the application presenting/displaying thepaginated digital content) or until the paginated digital content is nolonger being displayed. Once the annotation mode has been invoked, themethod continues by determining 204 if an input has beenperformed/provided to create an annotation. Note that in someembodiments, the currently presented/displayed page may be locked whenthe annotation mode is invoked. Annotations may include highlights (asvariously described herein), which may be created by dragging over thetext (e.g., as shown in FIGS. 6 a-d), and notes (as variously describedherein), such as sticky notes, which may be created by tapping on orselecting the currently presented/displayed page (e.g., as shown inFIGS. 7 a-d). If input has been performed to create an annotation, themethod continues by creating 205 the annotation.

If input has not been performed to create an annotation, the methodcontinues by determining 206 if annotations have been dragged togetherto merge them. Such a determination may include, in some embodiments,preliminarily determining if more than one annotation ispresent/displayed on the current page. In other embodiments, annotationsmay be able to be dragged across pages to merge them, such that apreliminary determination may be made to determine if more than oneannotation has been created in the entirety of the paginated digitalcontent. If annotations have been dragged together to merge them, thenthe method continues by merging 207 the annotations. For example, FIGS.10 a-d show an example of a sticky note being merged with another stickynote and FIGS. 11 a-d show an example of a highlight (including ahighlight note) being merged with another highlight (including its ownassociated highlight note). Note that in some embodiments, performing aninput to initiate merging annotations (e.g., a press-and-hold input on asticky note) may invoke the annotation mode (or highlight mode or notemode), for example.

Continuing from diamond 206 of FIG. 15, if annotations have not beendragged together to merge them, then the method can continue bydetermining 208 if a deletion input has been performed/provided todelete an annotation. Such a determination may include, in someembodiments, preliminarily determining if an annotation ispresent/displayed on the current page. If a deletion input has beenperformed to delete an annotation, the method continued by deleting 209the annotations. Example deletion input includes a press-and-hold on anannotation to grab it followed by a flick of the grabbed annotation(e.g., as shown in FIGS. 12 a-e) and/or a selection of an associated Xor garbage can deletion graphic (e.g., as shown in FIGS. 11 g-h). Ifannotations have been merged 207 or an annotation has been deleted 209,the annotation mode may be configured to determine 210 if an undo inputhas been performed to undo the merge or deletion function. The undoinput may include a shake input (e.g., shaking the computing device), aZ-gesture input, a Ctrl+Z command, or selecting an undo button from adialog (which may pop up in response to merging or deletingannotations), just to name a few examples. If an undo input has beenperformed after a merge or delete function has been performed, themethod continues by undoing 211 the annotation merge/deletion. In someembodiments, the annotation mode may be configured to allow undoing anynumber of merges/deletions as long as no intervening non-merge/deleteactions (e.g., create an annotation) have been performed.

If a deletion input has not been performed, the method continues toreview for an input (e.g., create annotation input or merge annotationsinput) until the annotation mode is exited, turned off, or no longerinvoked. In some embodiments, once the annotation mode is invoked, theorder of reviewing for annotation creation input, merging input, ordeletion input may be varied (e.g., review for such input may happenconcurrently). In some embodiments, the annotation mode may also beexited, turned off, or no longer be invoked if another mode isselected/invoked, if the paginated digital content is closed, and/or ifa home or power button is selected, just to list a few possibleexamples.

Numerous variations and embodiments will be apparent in light of thisdisclosure. One example embodiment of the present invention provides anelectronic device comprising a display for presenting paginated digitalcontent to a user, and a user interface including an annotation mode.The annotation mode allows a user to add annotations in the form ofhighlights and/or notes to paginated digital content, wherein anannotation can be merged with another annotation in response to at leastdragging and dropping one of the annotations on the other annotation tocreate a merged annotation. In some cases, notes can be merged inresponse to dragging and dropping a first note on a second note tocreate a merged note. In some such cases, the merged note inherits colorand location properties from the second note, while in some such cases,the merged note includes both the first and second note contents and thecontents from the first note are added to the contents of the secondnote. In some cases, highlights can be merged in response to dragging afirst highlight over at least a portion of a second highlight and thenreleasing the drag to drop the first highlight on the second highlightto create a merged highlight that highlights content from both the firstand second highlights. In some such cases, the merged highlight inheritscolor from the first highlight. In some such cases where highlights canbe merged using a drag and drop input, the first highlight includeshandles on either end of the highlight that allow the highlight to bedragged to expand or contract the highlight from either end. In somesuch cases where highlights can be merged using a drag and drop input,the first highlight and/or the second highlight include an associatednote, such that when the first highlight is merged with the secondhighlight, the merged highlight includes an associated note includingcontents of a note associated with the first highlight and/or a noteassociated with the second highlight. In some cases, a first note can bemerged with a second note that is associated with a highlight, inresponse to at least dragging and dropping the first note on either thesecond note or the highlight with which the second note is associated tocreate a merged note associated with the highlight. In some cases, anote can be merged with a highlight in response to at least dragging anddropping the note on at least a portion of the highlight to create amerged annotation. In some cases, a highlight can be merged with a notein response to at least dragging and dropping the highlight on at leasta portion of the note to create a merged annotation. In some cases, themerging of the first annotation and the second annotation can be undoneusing an undo input. In some such cases, the undo input includes a shakeinput and/or a Z-shaped gesture input.

Another example embodiment of the present invention provides a mobilecomputing device comprising a processor, a touch screen display fordisplaying paginated digital content to a user and allowing user input,and a user interface executable by the processor and including anannotation mode. The annotation mode allows a user to add annotations inthe form of highlights and/or notes to paginated digital content,wherein a first annotation can be dragged in response to apress-and-hold input performed on the first annotation and wherein thefirst annotation can be merged with a second annotation in response todragging and dropping the first annotation on the second annotation tocreate a merged annotation. In some cases, visual, aural, and/or tactilefeedback is provided in response to the press-and-hold input performedon the first annotation to indicate that the first annotation can bedragged, and/or in response to dragging and dropping the firstannotation on the second annotation to indicate that the annotationshave been merged.

Another example embodiment of the present invention provides a computerprogram product comprising a plurality of instructions non-transientlyencoded thereon that when executed by one or more processors cause aprocess to be carried out. The computer program product may include oneor more computer readable mediums such as, for example, a hard drive,compact disk, memory stick, server, cache memory, register memory,random access memory, read only memory, flash memory, or any suitablenon-transitory memory that is encoded with instructions that can beexecuted by one or more processors, or a plurality or combination ofsuch memories. In this example embodiment, the process is configured to,in response to a press-and-hold input performed on a first annotationpreviously added to paginated digital content, allow the firstannotation to be dragged, wherein the first annotation is a highlightand/or a note. The process is also configured to, in response todragging and dropping the first annotation on a second annotationpreviously added to the paginated digital content, merge the first andsecond annotations to create a merged annotation, wherein the secondannotation is a highlight and/or a note. In some cases, the process isfurther configured to undo the merging of the annotations in response toan undo input. In some such cases, the undo input includes a shake inputand/or a Z-shaped gesture input. In some cases, the process is furtherconfigured to present a dialog requesting to undo a merging ofannotations in response to merging annotations. In some cases, theprocess is further configured to provide visual, aural, and/or tactilefeedback in response to the press-and-hold input performed on the firstannotation to indicate that the first annotation can be dragged, and/orin response to dragging and dropping the first annotation on the secondannotation to indicate that the annotations have been merged.

The foregoing description of the embodiments of the present disclosurehave been presented for the purposes of illustration and description. Itis not intended to be exhaustive or to limit the disclosure to theprecise forms disclosed. Many modifications and variations are possiblein light of this disclosure. It is intended that the scope of thepresent invention be limited not by this detailed description, butrather by the claims appended hereto.

What is claimed is:
 1. An electronic device, comprising: a display forpresenting paginated digital content to a user; and a user interfaceincluding an annotation mode, the annotation mode allowing a user to addannotations in the form of highlights and/or notes to paginated digitalcontent, wherein an added annotation can be merged with another addedannotation in response to at least dragging and dropping one of theadded annotations on the other added annotation to create a mergedannotation.
 2. The device of claim 1 wherein notes can be merged inresponse to dragging and dropping a first note on a second note tocreate a merged note.
 3. The device of claim 2 wherein the merged noteinherits color and location properties from the second note.
 4. Thedevice of claim 2 wherein the merged note includes both the first andsecond note contents and wherein the contents from the first note areadded to the contents of the second note.
 5. The device of claim 1wherein highlights can be merged in response to dragging a firsthighlight over at least a portion of a second highlight and thenreleasing the drag to drop the first highlight on the second highlightto create a merged highlight that highlights content from both the firstand second highlights.
 6. The device of claim 5 wherein the mergedhighlight inherits color from the first highlight.
 7. The device ofclaim 5 wherein the first highlight includes handles on either end ofthe highlight that allow the highlight to be dragged to expand orcontract the highlight from either end.
 8. The device of claim 5 whereinthe first highlight and/or the second highlight include an associatednote, such that when the first highlight is merged with the secondhighlight, the merged highlight includes an associated note includingcontents of a note associated with the first highlight and/or a noteassociated with the second highlight.
 9. The device of claim 1 wherein afirst note can be merged with a second note that is associated with ahighlight, in response to at least dragging and dropping the first noteon either the second note or the highlight with which the second note isassociated to create a merged note associated with the highlight. 10.The device of claim 1 wherein a note can be merged with a highlight inresponse to at least dragging and dropping the note on at least aportion of the highlight to create a merged annotation.
 11. The deviceof claim 1 wherein a highlight can be merged with a note in response toat least dragging and dropping the highlight on at least a portion ofthe note to create a merged annotation.
 12. The device of claim 1wherein the merging of the first annotation and the second annotationcan be undone using an undo input.
 13. The device of claim 12 whereinthe undo input includes a shake input and/or a Z-shaped gesture input.14. A mobile computing device, comprising: a processor; a touch screendisplay for displaying paginated digital content to a user and allowinguser input; and a user interface executable by the processor andincluding an annotation mode, the annotation mode allowing a user to addannotations in the form of highlights and/or notes to paginated digitalcontent, wherein a first annotation can be dragged in response to apress-and-hold input performed on the first annotation and wherein thefirst annotation can be merged with a second annotation in response todragging and dropping the first annotation on the second annotation tocreate a merged annotation.
 15. The device of claim 1 wherein visual,aural, and/or tactile feedback is provided: in response to thepress-and-hold input performed on the first annotation to indicate thatthe first annotation can be dragged; and/or in response to dragging anddropping the first annotation on the second annotation to indicate thatthe annotations have been merged.
 16. A computer program productcomprising a plurality of instructions non-transiently encoded thereonthat when executed by one or more processors cause the following processto be carried out: in response to a press-and-hold input performed on afirst annotation previously added to paginated digital content, allowthe first annotation to be dragged, wherein the first annotation is ahighlight and/or a note; and in response to dragging and dropping thefirst annotation on a second annotation previously added to thepaginated digital content, merge the first and second annotations tocreate a merged annotation, wherein the second annotation is a highlightand/or a note.
 17. The computer program product of claim 16, wherein theprocess is further configured to undo the merging of the annotations inresponse to an undo input.
 18. The computer program product of claim 17,wherein the undo input includes a shake input and/or a Z-shaped gestureinput.
 19. The computer program product of claim 16, wherein the processis further configured to present a dialog requesting to undo a mergingof annotations in response to merging annotations.
 20. The computerprogram product of claim 16, wherein the process is further configuredto provide visual, aural, and/or tactile feedback: in response to thepress-and-hold input performed on the first annotation to indicate thatthe first annotation can be dragged; and/or in response to dragging anddropping the first annotation on the second annotation to indicate thatthe annotations have been merged.